AppVeyor build 1.0.4002

AppVeyor build 1.0.4004 for commit d97e5f6 is now complete.

• manuscript-1.0.4004-d97e5f6.html
• manuscript-1.0.4004-d97e5f6.pdf
• spelling-errors-1.0.4004-d97e5f6.txt
• spelling-error-locations-1.0.4004-d97e5f6.txt

Thank you so much for working on this @agitter! I think as long as we are getting the markdown document into something that is vaguely compatible with the template, it will be a lot easier to clean up the template. I'm not sure how well I can review this PR because it is really complex. @mprobson has also said he can take a look, but I imagine he'll have the same issue with not quite understanding the under-the-hood of Manubot well enough. I can definitely help with reviewing the outputs, though.

I'm happy to walk through some of the build script changes if you comment on anything you're curious about. The build.sh script is a little messy because I copied a large code block while trying to debug the tex export.

The main issue is that the newer version of Pandoc does make the HTML and PDF outputs look different. A short term workaround would be to create two conda environments and manually toggle between them when we're preparing LaTeX export versus making general builds the rest of the time. It would be better to fix the formatting issues, but we have so little time.

For review, you can ignore all the build/assets/acmart-master files. Those are just the files I downloaded from ACM. The main focus of the review (where @mprobson may be able to help) is the two attachments in my original post. We should identify any major problems in the exported .tex file that I can try to fix (e.g. authors) so we can move on to the content.

@agitter this is an amazing effort! I looked through the committed files and I think I understand most of what they're doing. I'll try to chime in with what I can parse but apologies if I missed something.

As far as I can tell the generated .bib file looks correct.

We should be able to generate a rough pdf in the file stage of the build.sh file using one of two methods, depending on what's available in the CI environment:

latexmk -pdf output/$INDIVIDUAL_KEYWORD-manuscript.tex && latexmk -pdf -c output/$INDIVIDUAL_KEYWORD-manuscript.tex

or

pdflatex output/$INDIVIDUAL_KEYWORD-manuscript.tex &> /dev/null
bibtex output/$INDIVIDUAL_KEYWORD &> /dev/null  # not sure about this line if we're using natbib
pdflatex output/$INDIVIDUAL_KEYWORD-manuscript.tex &> /dev/null
pdflatex output/$INDIVIDUAL_KEYWORD-manuscript.tex &> /dev/null

I use the first command locally to build my finished pdfs (and clean the outputs, everything after the && is just housekeeping) and the seconds set of commands is from this blog post, which links to bibtex's instructions. We may even be able to use pandoc, e.g. pandoc methods.md -o methods.pdf and potentially skip the .tex and .bib file generation step.

Also, I don't know that all the build/assets/acmart-master/* files are necessary. The acmart style should be available by default depending on the environment and LaTeX installation. If not, we can probably install it via whatever LaTeX distribution we have. I think we'd only want to include the .cls file (and not everything else) if none of that works.

The majority of my observations are about the generated .tex file.

1. We should have a method to manually specify the style as docmentclass: acmart and classoptions: sigconf (see below).

2. In the .tex file I expected the bibliography to look something like:

\bibliographystyle{ACM-Reference-Format}
\bibliography{methods.bib}

The linked Pandoc-to-LaTeX template does have functionality to generate this but I don't see it being employed even though I see the natbib option in latex.yaml.

3. In general, it looks like Pandoc expects a yaml stub at the top of the file to fill in all the variables. It appears we can generate and provide this separately via the metadata-files option, e.g. the list of authors and per paper specific LaTeX/YAML configurations, including the document class and bibliography style options mentioned above.

4. I think we should consider maintaining (or forking) our own template file. You linked several good examples and I'll add one more here: acm-pandoc-conf.tex from this (very helpful) blog post. I can try and create a minimal working example if that would be helpful although I am still trying to figure out how to test the Manubot pipeline.

Thanks for reviewing. You have a good understanding of what's going on.

We should be able to generate a rough pdf in the file stage of the build.sh file

This should be possible, but I anticipate we'll run into a lot of minor details when trying to fully automate pdf generation that will slow us down. manubot/rootstock#249 and manubot/rootstock#256 have additional context about building the pdf in the continuous integration environment with pandoc (the old Travis CI one, which was less flexible).

The workflow I had in mind was

• automatically generate .tex and .bib source files
• (optional) run a simple script to fix some .tex file issues via string replacement
• download and manually polish the .tex to fix issues noted above and others
• locally build a pdf

What do you think about that?

I don't know that all the build/assets/acmart-master/* files are necessary

I can delete these and keep only the .cls file. My goal was to provide what is needed for someone building locally.

In general, it looks like Pandoc expects a yaml stub at the top of the file to fill in all the variables.

Yes, Manubot has a different convention for setting authors, but we can modify the metadata in the yaml stub or pass metadata to pandoc as a command line argument. I'm passing the title via the command line for individual manuscripts to override the title in the yaml block. Author metadata is probably worth automating before we merge.

I am still trying to figure out how to test the Manubot pipeline.

To run this locally

• Create and activate the conda environment (instructions) using the updated environment.yml from this branch
• From the root directory of the repository run BUILD_HTML=false BUILD_PDF=false BUILD_INDIVIDUAL=true build/build.sh (save time by not building the full manuscript, only the methods manuscript)

If I'm forgetting a step, I can help debug. I agree that we would need to fork a template if we want to perfect the automated build. If we instead can settle with a .tex file that has content that can by copied into the ACM template, we can live with a lot of errors in the template.

Edit: I realized every time I change the gist you need to update the metadata link to it.

That's not a problem for me. I'm editing these files regularly anyway. We can integrate it into this repo after we merge this pull request.

I'll work on the conflicts statement after pushing some text edits to #947.

Just pushed a change to the gist enabling bibfile which automatically adds the right bibliography line. Now we just need to automate removing the CSLreference section.

I noticed we need the ACM copyright info. I can hardcode but the template could ingest the following (I'm just not sure where to check it in):

---
acm:
- copyrightyear: 2021
  copyright: acmcopyright
  conference: ACM-BCB '21
  conferencetitle: ACM-BCB '21: ACM Conference on Bioinformatics, Computational Biology, and Health Informatics
  date: August 01--04, 2021
  location: Online
...

Outstanding things on my TODO list (I think this all needs to be done manually?):

• Move front matter into proper \begin{}...\end{} sections (may need to do by hand)
  • Abstract
  • Keywords
  • CCS
• Remove section number from acknowledgements (again, may be by hand)

@mprobson I usually manually change the acknowledgements, do you want me to send you the correct ones for this paper?

Edited to add: have to go into a meeting so doing this here just in case--

We are grateful to Josh Nicholson and Milo Mordaunt for their support with the scite plugin, and to David Nicholson for the suggestion and feedback to enable the reporting of the locations of spelling errors in the spell-checker tool. We thank Nick DeVito for assistance with the Evidence-Based Medicine Data Lab COVID-19 TrialsTracker data.

I can hardcode but the template could ingest the following

I'll update to add that block into the YAML I generate.

Minor / Unimportant notes

I can fix those.

AppVeyor build 1.0.4065 for commit f7780b8 is now complete.

• manuscript-1.0.4065-f7780b8.html
• manuscript-1.0.4065-f7780b8.pdf
• spelling-errors-1.0.4065-f7780b8.txt
• spelling-error-locations-1.0.4065-f7780b8.txt

I usually manually change the acknowledgements...

That's great, thanks! I think the bigger issue is that we need to use \section*{Acknowledgments} and I'm not sure how to automate that.

I pushed changes that resolve most/all of the issues above:

• clean up the included ACM files
• only show the CCS concepts section when exporting as tex
• add the ACM metadata and conflicts statement for the template
• suppress the references section in the markdown
• exclude 70.coi-contribs.md (conflicts table, contribution table, acknowledgements)
• add a new markdown file for methods acknowledgements and make the section unnumbered

The acknowledgments section now shows up as

\hypertarget{acknowledgements}{%
\section*{Acknowledgements}\label{acknowledgements}}
\addcontentsline{toc}{section}{Acknowledgements}

@rando2 you can edit the content in 61.methods-ack.md if you need to edit the acknowledgements text for this individual manuscript.

I'm happy with this tex output. It's a lot cleaner than I would have guessed. From here, should we create a new issue with a single checklist of manual changes needed before submitting?

Latest version:
methods-manuscript.pdf
methods-manuscript.tex.txt

I just realized we don't have a funding statement anywhere. Should we add that to 61.methods-ack.md? I think that would be best in terms of making it quickly and saving space.

Wow, that's amazing! I agree re: funding statement. I usually put my sources in the acknowledgements.

I'm currently getting a weird error trying to build the latest commits:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
KeyError: 'commit'

I suspect that something has gone wrong with my python but wanted to confirm it's working for you.

I suspect that something has gone wrong with my python but wanted to confirm it's working for you.

Strange. It is working for me. Is that error coming from Manubot or my script?

It's definitely a system configuration issue on my end, although I am not sure what changed... I have some weird combination of Tmux, Conda, and MacOS... I re-ran on a known good commit and it still fails. conda deactivate/activate doesn't fix it nor does destroying and recreating the environment. I imagine it's just something I need to debug on my own.

Here's the command plus error:

(manubot) C02YD02QJGH8:covid19-review mrobso01$ BUILD_HTML=false BUILD_PDF=false BUILD_INDIVIDUAL=true build/build.sh
Traceback (most recent call last):
  File "<string>", line 1, in <module>
KeyError: 'commit'

I think I figured out what's happening... I seem to be dying on this line:

EXTERNAL_RESOURCES_COMMIT=$(curl -sS https://api.github.com/repos/greenelab/covid19-review/branches/external-resources | python -c "import sys, json; print(json.load(sys.stdin)['commit']['sha'])")

@rando2 and I share an IP address currently and she's getting rate limited by the GitHub API so I suspect I am now being throttled too...

Update: Connecting to my VPN resolves the issue!

AppVeyor build 1.0.4083 for commit 3baa06f is now complete.

• manuscript-1.0.4083-3baa06f.html
• manuscript-1.0.4083-3baa06f.pdf
• spelling-errors-1.0.4083-3baa06f.txt
• spelling-error-locations-1.0.4083-3baa06f.txt